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Remarks 

Claims l-M ate pending in the application Claims i-H ate rejected All 
rejections at e respect fit ! ! \ tra\ ersed 

Flic Fxammei te]ected claims i-M nndci 35 (JSC 112 first paragiaph 
I he F\ammer states that (he specification does not enable "one of ordinary 
skill offhe at I to lecogtn/e the fealuio 'Mdeo object planes " 

With all due respect, the F\ammet "s rejection is nnpioper First, the 
application does identify "video object planes " 

"Newer video coding standards, such as MPEG --4, see 'Information 
Tcchnologv — Genetic coding of audioA tsual objects," ISO 1FC FfMS 
144%-2 (MPFG4 Visual). Nov 10<>8. allow arbitrary-shaped objects to be 
encoded and decoded as sepatate video objeti planes (YOP) This emerging 
standard is intended to enable multimedia applications, such as interactive 
Mdeo, where natural and synthetic materials ate integrated, and where access 
is universal Fot example, one nnyht want to "cut-and-pasUf* a moving 
figure 01 object ftom one video to another in (his type of scenario, it is 
assumed dial the objects ui the multimedia content have been identified 
through some type of segmentation aigotithm 

The application also goes on to site U.S Patent Application Sn. 09 32o,~50 
"Method for Ordering image Spaces to Search for Object Surfaces" filed on 
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June 4, IW b> Lin ei ai. now U N Patent 6.400.846 There, V()P\are 
sufficient]} anddearh desenbed 

"With VOPs, each image of a video sequence is segmented into aibitumh 
shaped image regions fach \'OP deseubes audeo object m terms of, fot 
example, shape, motion, and te\ture Hie exact method of producing 
VOP's fiom the source imageiv is not defined b> the siandaida it is 
assumed that "naliual" objects ate repiesented In shape information, ni 
addition to the usttal luminance and ehiommance components Shape data 
can be provided as a segmentation mask, oi as a gray scale alpha plane to 
represent multiple overlaid objects." 

The above would make it clear to one of ordinary skill in the art what a 
video object plane is. 

Second, under M h F P 2104 Otye) hxampies of ruablement Issues. "To 
establish a teasonable basis for questioning the adequacy of a disclosure, the 
exam mot must present a factual anahsis of a disclosure to show that a 
person skilled in the art would not be able lo make and use the claimed 
it mention without icsottiiig to undue experimentation lu the rejection, the 
hxaimnei does not pio\ide (he requited factual anahsis Use b,xammci 
simph states that he could not find the "video object plane" feature in the 
specification Not being able to find the claimed feat nit, is not a factual 
anahsis \\ ith all due icspect. the above will hopeiulh assist the hxaimner 
now to find the VOP featuies 
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Hind, the above section oil he M P F P \v itfi i expect lo the refei cueing pnoi 
at I document states that the anafvsis would appear to he less eutieal wheie 
the features, die essentially standard components CeitauiK, the VOPs i\$ 
claimed aie defined in gteat detail t« the referenced standards and other 
Cited pilot art documents, and would not require undue experimentation b\ 
one of ordmarv skill in the ait 

1 he invention pio\ kIcn a method foi deieniiirnng similanties of 
mteiptetauon between portions of multimedia (videos) at a \erv high level, 
e g , sumfai action in an advemuie movie seoung oppottumties in a sports 
video, lomantie activitv in a gothic movie, flight m a honoi mo\ic humoi m 
a coined) nun ic. and so forth 1 lie term high-lev el k is used because the 
snnilantv cousideis a sequence of semanlic e\ents extended o\er a icSameh 
long tune period. 

Therefore, the invention segments multimedia content to extract video 
objects in the form of video object planes, which can encode arbitrary- 
shaped objects according to the MPEG-4 standard, also known as li.264 or 
AVC. see Specification, page 2: 

"Newer video coding standards, such as M.PEG-4, see 'In formation 
Technology — Generic coding of audio/visual object s," ISO/1 EC FDIS 
14496-2 (MPEG-4 Visual ), Nov. 1.998, allow arbitrary-shaped objects 
to be encoded and decoded as separate video object planes ( VOP}... 
The most recent standardization effort taken on by the MPEG 
committee is that of MPEG-7, formally called ^Multimedia Content 
Description Interface," see "MPEG-7 Context, Objectives and 
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leehmeal Roadmap" ISO \hC N2~29, March 1<>99 hs a cntiall>, tins 
standard plans to nieorpoiate a set of descriptors and description 
schemes dial can be used to describe various types of multimedia 

content." 

In the art, these newer, higli-le\el structures are distinguished from older,, 
hnv-ieve! features such as color and motion 

Claims 1-13 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Yeo et al., U.S. Patent No. 5,821,945 (Yeo), in view of Puri et al. 

Video object planes f VOW are defined in she 1 1 204/MP1.G-4 oi WC 
standards ihe call lor proposals was in Mas 1^98. and die first draft design 
foi the new standard was not adopted in until 1999 I lie Yeo patent 
application was tiled m May P>9~ Yeo could not have known about video 
object planes as claimed 

The invention segments multimedia content to extract v ideo object planes 
The decomposition of videos into a hietaiclticaf scene transition graph 
according to Yeo icfleets acts, scenes and shots of the v ideo. not v ideo object 
planes. 

Yeo does not extract and associate features of the video object planes to 
produce content entities. Instead, the browsing process of Yeo is "automated 
to extract a hierarchical decomposition of a complex video selection i n four 
steps: the identification of video shots, the cl usteri ng of video shots of 
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similar \LiUfd contents, ihe presentation of ihe content and structure to the 
users \ia the scene tiansition giaph, and ImalK the lueratehieal organization 
of {he graph st incline " 

Yeo does not measure lugh-le\ef temporal attributes of eaeh content entm 
Yeo states: 

"I o\\ level vision analyses operated on \ideo frames aeihe\e 
reasonably good results for the measurement of similarity {or 
dissimilarity ) of different shots Similarity measures based on image 
attributes such as color, spatial correlation and shape can distinguish 
different shots to a significant degree, e\en when operated on much 
reduced images as the DO images Bosh color and simple shape 
information are used to measure similarity of Ihe shots." 

ihe ptoblenis with lo\\~le\el features as in Yeo are distinguished in the 

present application at pages 2 and 3. 

"Anothei problem with such tow -lex el desenptots, in genet aL is that a 
high~le\el interpretation of the object or multimedia content is 
difficult to obtain Hence, theit is a limitation in the 1e\d of 
repiesensahon To overcome she drawbacks mentioned abuse and 
obtain a higher-le\el of representation, one may consider more 
elaborate dcsenption schemes that combine several low-le\d 
descriptors. In fact, these description schemes may e\en contain other 
description schemes, see "MPHCi-7 Description Schemes {\ 0.5)," 
ISO ID/N2844, July 
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Yco does not describe content entities and comparing the temporally ordered 
content entities m a plmafitv of the directed acyclic staphs to determine 
similar interpretations of the multimedia content 

I he JXannner states that Yeo does not describe video object planes, and 
cites Pun Pun cannot be combined with Yeo 1 lie Examiner's teaaon that 
(his would provide "a system foi description of scene m a huh flexible 
manner" snakes no sense 1 he m\ ention is not concerned with desci ihmg 
scenes The invention claims oidenng multimedia content 

The nnenlion measines atti ibutc=> of content entities that include mtensiH 
attributes ihe iAannner again enoiieoush cites Yen. at column 7, hues 35 
et seq , measines correlations between frames as UiSTeienees 
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twkfkt. tlx s}os ^ n^m,,. in „0*.i& » l*K^ *m, t»K t.>>U pi U- 

r\v««u e>\"ict<r ><r it-- ps; *\ ' n^i^nc*. \< ojiUfor; AlicivO 



.lUi -v tiX^ !» KM 'fctt. Vli limit 1 r^'l. ! il3l* i , 



^ 'UKltH ' IMtH\.tw<li this 1» " tl ) 



There is nothing in the above about intensity attributes. Clarification is 
respectfully requested to exactly point out where Yeo describes measuring 
intensities as claimed. 
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The Examiner keeps reciting the same sections, yet as repeatedly point out, 
an extensive word search of Yeo easily reveals {.hat the words "intensity" or 
"intensities" never appear anywhere in Yeo. The Examiner's rejection of 
claim 2 continues to be improper. 



Yeo does not measure attributes of content entities that include direction 
attributes. 

>kK n <,t men iu> k 



The section cited by the Examiner deals with shape. 



Yeo measutes. in images, the ''two-dimensional moment invariant ofthe 
lummance Those of otdmary skill in the art would not confuse direction 
and lu m i nance, see col um n 7 i i nes 13-19. An e\t en si v e w ord search of Yeo 
easilv reveals that the words "\inection" never appear an\ where in Yeo 
Ihe Lxammer's rejection of claim 3 continues to be improper 



Yeo does not measure attributes of content entities that include spatial 
attributes and the order is spatial 1 he Yeo measurements take place on 
frames. 
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Yeo does not rank order attributes measured of content entities. 



The scene transition graph in. Yeo is not derived form video object planes. 



There is nothing in column 19 that would indicate that: Yeo generates a 
summary of a video. 
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nmnher t>fchs tr* ik'-v^wi or tu <*o* „ »m tlx diwm* ,u»ty 
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Ct>luni»s 6 though 8 also do not ik\scnbe a \ idoo summary Applicants 
lc^peclfulK icq nest the 1 Auimnci to pomi out which \\ord(s* in Yeo mean **a 
\ ideo siunniary " 1 he Applicants ha\ e caiefully tead Neo but cannot find any 
video summarization steps. 



A word search reveals that the word "summary" appears exactly once in 
Yeo: 

"SUMMARY OF THE INVENTION " 



At column 7, Yeo states: 
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Ik fnxfU sysicn ixs as aootlu measure oi M f mhn;\ 

kiweer luo nugestne t^o-dimcnMot:^ ntnii cut .mx.au 
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ordei oi magnitude in alktcnt uo-nut munarl wr\ 

pu\\\ : lr man) cvmiples *h* m!k> ->1 tirv* moirctit *n\ r..n 

to ik lii.'d or lourlh rnmm* invaniiH can \« r\ :^ siurJ 

(>aicr> el magnitude 

There is absolutely nothing there that would suggest that a measure of 
similarity between two-dimensional luminance would suggest a three 
dimensional video A three dimensional video is a video that also includes 
depth, such as a MR1 video or CAT scan 

Claimed are directed ac> die graphs where nodes represent the content 
entities and edges represent breaks in the segmentation, and the measured 
attributes are associated with the corresponding edges Yen teaches a graph 
'"with nodes representing scenes and edges representing the progress of the 
story trom one scene to the next " 

Claimed is at least one secondary content entity associated with a particular 
content entity, and wherein the secondary content entity is selected during 
the traversing Nowhere in columns 2-6 are these limitations described 

Claimed is a summary of the multimedia with a selected permutation of the 
content entities according to the associated ranks At columns V- 
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Yeo allows the user to rearrange nodes in a graph. There is nothing there to 
indicate that: content entities can be permuted according to rank. 

It is believed that this application is now in condition for allowance. A 
notice to this effect is respectfully requested. Should further questions arise 
concerning this application, the Examiner is invited to call Applicants' 
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attorney at the number listed below. Please charge any shortage in fees due 
in connection with the filing of this paper to Deposit Account 50-0749. 



Respect fulls siibmilted 

Mitsubishi lilectne Research Laboratories, Inc. 

By 

/Dirk Brinkman/ 



201 Broadway, 8* Floor 

Cambridge, MA 021 39 
Telephone: (617)621-7539 
Customer No. 022199 



Dirk Brinkman 
Attorney for the Assignee 
Reg. No. 35,460 



17 



